Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness In this work, we improve robustness-aware self-supervised pre-training by learning representations that are consistent under both data augmentations and adversarial perturbations. Our approach leverages a recent contrastive learning framework, which learns representations by maximizing feature consistency under differently augmented views. This fits particularly well with the goal of adversarial robustness, as one cause of adversarial fragility is the lack of feature invariance, i.e., small input perturbations can result in undesirable large changes in features or even predicted labels. We explore various options to formulate the contrastive task, and demonstrate that by injecting adversarial perturbations, contrastive pre-training can lead to models that are both label-efficient and robust. We empirically evaluate the proposed Adversarial Contrastive Learning (ACL) and show it can consistently outperform existing methods. For example on the CIFAR-10 dataset, ACL outperforms the previous state-of-the-art unsupervised robust pre-training approach by 2.99% on robust accuracy and 2.14% on standard accuracy. We further demonstrate that ACL pre-training can improve semi-supervised adversarial training, even when only a few labeled examples are available.

Strengths: The paper's main idea is easy to follow: extending a recently successful contrastive learning framework SimCLR [2] to adversarial training. While SimCLR is already popular for a number of tasks, exploring its usage for adversarial defense appears to be new and original. The authors explained why SimCLR might be particularly suitable for the goal of adversarial robustness: one cause of adversarial fragility is the lack of feature invariance to small input perturbations, and SimCLR learns representations by maximizing feature invariance under differently augmented views. That makes this paper well motivated and grounded. The main technical part of this paper explores options to formulate the contrastive task.

This paper focuses on adversarial training. The proposal is to incorporate adversarial training into the pre-training step, which makes the pre-training techniques even more robustness-aware. This can be seen as an extension of SimCLR (with the incorporation of adversarial training). The philosophy behind sounds quite interesting to me, namely, introducing adversarial robustness into self-supervised learning and formulating the contrastive task. This philosophy leads to a novel algorithm design I have never seen, i.e., Adversarial-to-Adversarial (A2A), Adversarial-to-Standard (A2S), and Dual Stream (DS).

Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness In this work, we improve robustness-aware self-supervised pre-training by learning representations that are consistent under both data augmentations and adversarial perturbations. Our approach leverages a recent contrastive learning framework, which learns representations by maximizing feature consistency under differently augmented views. This fits particularly well with the goal of adversarial robustness, as one cause of adversarial fragility is the lack of feature invariance, i.e., small input perturbations can result in undesirable large changes in features or even predicted labels. We explore various options to formulate the contrastive task, and demonstrate that by injecting adversarial perturbations, contrastive pre-training can lead to models that are both label-efficient and robust. We empirically evaluate the proposed Adversarial Contrastive Learning (ACL) and show it can consistently outperform existing methods.